Camera and Sensor Controls for Remotely Operated Vehicles and Virtual Environments

ABSTRACT

A hand controller that includes a control member mounted on a hand held base for commanding a remote vehicle or movement of a virtual object within a virtual environment includes an input device mounted for generating tilt commands for a sensor mounted on the remote vehicle or the virtual object, the input device being located on a front of the base and oriented to be displaced up and down to tilt the sensor or the virtual point of view when the base is held during operation.

This application is a continuation-in-part of U.S. application Ser. No.16/114,190 filed Aug. 27, 2018, which is a continuation-in-part of U.S.application Ser. No. 15/964,064, filed Apr. 26, 2018, which is acontinuation-in-part of U.S. application Ser. No. 15/796,744 filed Oct.27, 2017, which claims the benefit of U.S. provisional application No.62/413,685 filed Oct. 27, 2016. The entirety of each of theseapplications is incorporated herein by reference for all purposes.

FIELD OF THE INVENTION

The present disclosure relates to controllers through which a personcommands movement of a remote target augmented with a camera and othersensors, such as camera-carrying aerial drones and remotely operatedunderwater vehicles.

BACKGROUND OF THE INVENTION

Input devices or controllers, such as, joysticks, control columns,computer mice, cyclic sticks, foot pedals generate control inputs for areal or virtual target by sensing movement of one or more controlmembers by a person that is commanding or controlling movement andoperation of a target. These types of controllers have been used tocontrol inputs for parameters such as to control pitch, yaw, and roll ofthe target, as well as navigational parameters such as translation(e.g., x-, y-, and z-axis movement) in a three-dimensional (3D) space,velocity, acceleration, and/or a variety of other command parameters.Examples of physical targets whose movement can be controlled remotelyinclude aircraft such as aerial drones, submersible vehicles, roboticarms, industrial cranes and spacecraft. Examples of virtual targetswhose movement can be controlled remotely include virtual and augmentedreality simulations, computer aided design of 3-D objects and a widerange of computer games.

Typical drone, ROV and gaming controllers have two joy sticks mounted toa hand-held base for generating inputs for commanding movement andoperation of a remote vehicle, as well as other inputs or controls forcontrolling operation of a camera on the remote vehicle.

SUMMARY

The present disclosure relates to improvements to controllers that areused by an operator to control movements of remote vehicles like aerialdrones or other unmanned aircraft and spacecraft, submersible vessels,robotic arms, industrial cranes and similar targets that are beingcontrolled remotely, and that have a steerable imaging sensor or othertype of steerable sensor on a motorized mount that allow for a leasttilting the sensor's orientation relative to the frame of reference ofthe remote target.

One aspect of the disclosure relates to orientation and positioning amechanical input device on a controller for steering a sensor mounted onthe target. Typical controllers for drones and other unmanned aerialvehicles of the type that are held by an operator have an off-axisrotary wheel that the operator turns to generate control inputs thattilt a video camera or other imagining sensor mounted on the drone. Therotary wheel is typically placed and oriented on the controller so that,if the operator's frame of reference is the drone's frame of reference,the wheel is rotated in plane that is orthogonal to the plane in whichthe camera tilts. It is common to forget which direction moves thecamera up and which direction moves it down while operating such acontroller. It is therefore relatively difficult to obtain imagery withconventional drone controllers owing to their non-intuitive inputinterfaces as well as the difficulty in readjusting movement rates whileactively flying and acquiring imagery. The mechanical input device isoriented and positioned on the controller so that displacement of theinput device corresponds to the movement of the sensor relative to thetarget when the controller is held in its normal operating position. Inone embodiment, to tilt a sensor up and down on a target, an inputdevice on the controller for generating a command or control input totilt the sensor is oriented on the controller so that its displacementin a vertical plane relative to the user or pilot when the controller isin its normal, operational position and operated by the user results ingenerating a command to the target to tilt the sensor in the directionin which it the input device is displaced.

Such a controller can be adapted for controlling virtual environments,including movement of a view angle or point of view (POV) within acomputer gaming or 3-D computer aided design (CAD) environment, as wellas in virtual and augmented reality simulations. In one example,displacement of the input device results in a secondary control systemmoving a POV or simulated camera view relative to an asset moving withina virtual environment under the control of an operator in a manner thatcorresponds to the direction of movement of the input device by theuser. Non-limiting examples of virtual environments include computergames, 3-D CAD, and virtual/augmented/mixed reality simulations.

A different aspect of the disclosure enables camera settings and speedadjustments for tilt and/or pan to be more easily adjusted “on the fly.”Typical commercial drone controllers, for example, require the use of anapplication running on a computer (laptop, tablet, smartphone or similardevice) to adjust camera and camera control settings. These settings areoften buried deep within a series of nested menus, which tend todiscourage their use, especially by those who are not expertvideographers. Most drones have a relatively short battery life, andsince difficult-to-access camera controls often involve stopping aflight so that the adjustments can be made, the amount of filming on anygiven flight can be quite limited.

In one representative embodiment, a rotary wheel or other input deviceon a base of controller allows toggling between functions to allow thecamera or other sensor parameters to be adjusted, for exampleresponsiveness of tilt and/or pan, shutter speed, exposure compensation,zoom and other functions that are otherwise typically only madeavailable using nested menus in an interface to a software programassociated with the controller.

Additional aspects, advantages, features and embodiments are describedbelow in conjunction with the accompanying drawings. All patents, patentapplications, articles, other publications, documents and thingsreferenced herein are hereby incorporated herein by this reference intheir entirety for all purposes. To the extent of any inconsistency orconflict in the definition or use of terms between any of theincorporated publications, documents or things and the presentapplication, those of the present application prevail.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a rear perspective view of an exemplary hand controllercapable of displaying target situational awareness information.

FIG. 1B is a front view of the hand controller of FIG. 1A.

FIG. 1C is a left side view of the hand controller of FIG. 1A.

FIG. 1D is a right side view of the hand controller of FIG. 1A.

FIG. 1E is a bottom, front right perspective view of the hand controllerof FIG. 1A.

FIG. 2A is a perspective view of a two-axis gimbal.

FIG. 2B is a cross-sectional view of the two-axis gimbal of FIG. 2Ataken along section lines 2B-2B.

DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

For promoting an understanding of the principles of the invention thatis claimed below, reference will now be made to the embodiments andexamples illustrated in the appended drawings. By describing specificembodiments and examples, no limitation of the scope of the claimedsubject matter, beyond the literal terms set out in the claims, isintended unless a definition for the terms is expressly given.Alterations and further modifications to the described embodiments andexamples are possible while making use of the claimed subject matter,and therefore are contemplated as being within the scope of the subjectmatter that is claimed.

In the drawings and description that follows, the drawings are notnecessarily to scale. Certain features of the invention may be shown inschematic form. Details or presence of conventional or previouslydescribed elements may not be shown in a figure in the interest ofclarity and conciseness.

The present disclosure describes several embodiments of a control systemor controller that allows a user to command using a single hand movementof a control target or point of reference (POR). The embodiments arerepresentative, non-limiting examples of controllers having one or morecontrol members that, when displaced by the user's hand or digits,generate a set of signals in each degree of freedom (DOF) of movement inwhich it is displaced. Each of these signals are then used to generatecontrol inputs that are transmitted to a target control system. Thecontroller maps the sensor signals to predetermined control inputs. Themapping can be, in one embodiment, changed or programmed so that thesignal from any degree of freedom being commanded with the controllercan be mapped to any control input for the target. Fewer than all of thedegrees of freedom of the one or more controllers can be used if notrequired for the application; and one or more of the degrees of freedomof movement of the one or more controllers can be locked againstmovement, either temporarily or permanently, if not required for anapplication for which the controller is being used.

The control member may be mounted to a base or platform held or worn bythe user, mounted on a tripod, or mounted on some other structure thatacts as a frame of reference for measuring displacement of the firstcontrol member. The controller further includes signal conditioningcircuits for interfacing sensors for measuring displacement of thecontrol members, a processor for running software programmed processes,such as those described herein, a battery or other source for power,interfaces for other hardware, and transmitters and receivers forwireless communication.

A mobile controller that is carried in one hand, with one or morecontrol members displaced by the other hand, provides a consistent,known reference frame (stabilized by the non-dominant hand) even whilemoving, e.g., walking, skiing, running, driving. For certain types ofapplications, for example inspection, security and cinematographic dronemissions, a control member may be mounted on a platform that can be heldor otherwise stabilized by the user's other hand. The platform mayinclude secondary controls and, if desired, a display unit. In oneexample, all 6-DOF inputs of a controller having first control memberwith 3-DOF of movement and a second control member mounted to it with anadditional 3-DOF of movement, can be reacted through the platform. Withsuch an arrangement, this example of a control system facilitatesmovement through the air like a fighter pilot with intuitive(non-deliberate cognitive) inputs.

FIGS. 1A-1E depict a non-limiting example of a controller 110 that isconfigured for commanding an aerial drone, such as a quadcopter, thoughit can also be adapted for controlling other types of targets that movein multiple degrees of freedom. Controller 110 has a first controlmember 112, a joystick-like structure with three independent degrees ofmovement that is intended to be gripped by a user's hand, a secondcontrol member 116 mounted on the first control member for manipulationby a thumb or other digit on the hand of the user that is gripping thefirst control member, and an optional third control member, which enablea user to generate four independent control inputs for commandingmovement of the drone in four degrees of freedom. A proximal end ofcontrol member 112 is pivotally connected to base 114 so that controlmember 112 can be independently pivoted along an x-axis andindependently pivoted along a y-axis. In this example, the base isconfigured to be held by a user's hand. A base held by a user's handprovides a consistent, known reference frame (stabilized by thenon-dominant hand) even while moving, e.g., walking, skiing, running,driving, can be used for inspection, security and cinematographic dronemissions.

Installed or mounted on the base are additional input devices that arenot used to command movement or orientation of the target, but othersystems associated with the target. These input devices each comprise acontrol member that can be pressed, rotated, deflected or shifted by afinger of a person in one or two degrees of freedom to operate thecontroller and to issue additional control inputs to the drone and itsvarious systems. The input devices include several buttons, dials,knobs, wheels and switches with one or two degrees of freedom. Toachieve two degrees of freedom, a dial, knob or wheel, for example, canbe depressed in addition to rotated. For example, flight sensitivityknob 130 sets the level of the sensitivity level of the movements of thehand controller movements by the user in order to increase or decreasethe responsiveness of the drone to the movements. Programable button 140that can be programed by the operator to perform a function desired bythe operator when depressed. A controller on/off switch or button 142 isused to turn to the power to the hand controller on or off. Arm dronebutton 144 enables the drone. Return home button 146 causes thecontroller to transmit a command to the drone that causes the drone toreturn to a predesignated place autonomously.

The base houses signal conditioning circuits for the input devices, aprocessor for running software programmed processes, a battery or othersource for power, interfaces for other hardware, and transmitters andreceivers for wireless communication. Actuation or displacement of theinput devices is sensed by circuits in the controller. In the case ofinput devices that are angularly displaced or rotated, the degree ofangular rotation is sensed. Power indicator light 148 illuminates whenthe hand controller is on, and battery indicator 150 indicates the levelof charge of the hand controller batteries.

Unmanned vehicles have a video camera for transmitting back to theremote operator. The video camera is mounted with the camera facing in aforward direction, so that when the vehicle moves forward the camera ispointing, usually, straight ahead in the direction of travel. The baseincludes manually-displaceable user interface elements for use by thecontroller to generate control inputs for a camera on the target,including a wheel 132 for changing camera settings, a paddle 134 forchanging camera tilt, a button 136 for releasing a camera shutter, andbutton 138 for starting and stopping recording of video.

A rotating wheel 132 changes camera settings for the video camera orother sensor input, such as exposure or sensitivity, aperture, andresolution, or to select preset combinations of these settings. Forexample, the rotary wheel has a plurality of positions that allows auser to rapidly reconfigure the camera-related setting by selecting fromseveral preconfigured settings. These parameters include theresponsiveness of the platform or mount on the remote vehicle tocommands to adjust (tilt and/or pan) the orientation or point of view(POV) of the camera (or sensor), or in the case of virtual object whosemotion in a virtual environment is being commanded, the virtual POV ofthe virtual object, as well as the ability to toggle to other commoncamera features without diving deep into an app, such as zoom, shutterspeed exposure compensation.

A button 136 for the shutter release causes a control input to begenerated and sent the drone to take a photograph or series ofphotographs. Button 138 starts and stops recording of the video sent bythe video camera. The video can be shown on a display (not shown) thatis attached to mount 152 and arm 154. In on embodiment, a smartphone isused for displaying video in addition to setting up and modifyingsettings for the controller, as well as performing additional processesfor the controller. The display or smartphone is connected by wire orwirelessly to the controller. In other embodiments, other displaydevices such as a tablet, computer, monitor, or standalone display couldbe substituted.

The base is of a size and shape that enables an operator or user to holdthe base in one hand and operate the control members with the otherhand. The placement of control members 116 and 118 onto the end ofcontrol member 112 allows an operator to operate control member 112 withthe palm and two or three fingers of one hand while at the same timeoperating control member 116 and control member 118 with the thumb andthe forefinger, respectively, of the same hand being used to operatecontrol member 112. The operator's other hand 133, can be placedunderneath the hand controller to support the hand controller and alsoto operate one or more additional inputs for controlling the camera withthe fingers of the hand that is supporting the hand controller. Thisallows the operator to keep one hand on the control member 112 at alltimes during controlled repositioning of the target to generate controlinputs in up to three degrees of freedom, while also being able tomanipulate control member 116 (and, optionally 118) to generate controlinputs for the target in additional degrees of freedom, hold the base,and operate a camera on the remote.

Aerial drones often have video cameras that offer tilting but notpanning the camera with respect to the drone. Adjustment of the view ofthe camera from side to side is typically accomplished by rotating thefacing of the drone. The drone is, in other words, rotated to pan thecamera. Therefore, this example includes paddle 134 for controllingcamera tilt that need only be deflectable in one degree of freedom.Displacement of the paddle 134 by deflecting it upwardly or downwardlycauses generation of a command input to the remote target's controlsystem that actuates a motor for a single axis camera tilt mechanism,with which a video camera is mounted to the drone.

The paddle 134 for commanding tilt is, in this example, located on thefront of base 114 where an index or other finger of a hand holding thebase can reach it. When an operator uses his or her hand to support base114, the operator can reach and operate camera rotary control 132 withthe thumb or forefinger of the same hand that is supporting base 114.The base is shaped to allow either right or left-handed use. Cameracontrols 136 and 138 for taking a photograph and starting and stoppingvideo recording are located on the side of the base where they can alsobe reached by the thumb or other fingers of the hand holding the base.

Deflection of paddle 134 within a plane that is vertical with respect tothe user's frame of reference when holding and operating the controllercauses the controller to generate a control output for changing cameratilt. Deflecting downwardly causes generation and transmission of acontrol output to the target's control system to tilt the cameradownwardly; deflecting it upwardly causes the controller to transmit acontrol output that results in the video camera being tilted upward.

In this example, the paddle 134 comprises part of an input device thatpivots about a single axis. The paddle is a control member that a usercan easily find by feel and deflect upwardly or downwardly. Deflectionof the control pivots a mount or support, the angular displacement ofwhich is sensed by the controller. The controller responds by generatinga camera tilt rate command and communicating it on a signal transmittedto the target, the command indicating the direction and amount or degreeof deflection. Alternatively, the controller issues an absolute command,meaning that the position of the input device corresponds to theposition of the camera, sensor or POV.

The magnitude of the deflection upward or downward of the input device134 determines the rate of adjustment that is communicated to thetarget's camera (or other sensor) control system, and the duration ofthe deflection determines the duration of the adjustment. In response toreceiving a tilt rate command, a camera control system on the targetoperates a motor that pivots or moves the camera mount to tilt thecamera in a direction, at a rate, and for a duration corresponding tothe direction, magnitude and duration of displacement from the nullposition of the input device 134.

In one embodiment, the paddle 134 is biased toward a center position inits range of motion, which corresponds to a “null” or zero inputposition. This allows it to be deflected by a user's finger upwardly ordownwardly and then released to return it to null or no-input by meansof a centering spring force. This is a natural, intuitive, andcomfortable movement for the user since upward motion corresponds to anupward adjustment in camera tilt and downward motion corresponds todownward adjustment of camera tilt. In an alternative embodiment, whengenerating an absolute command, the centering force may be omitted toallow the input device to remain in the position to which it is moved sothat the camera/sensor/POV will also remain at that position.

In alternate embodiments, other shapes or configurations of controlmembers could be substituted. For example, it could take the form of atoggle, wheel, or joystick that is mounted so that it is free to move ina vertical plane (with respect to a user when operating the controllerin its intended manner), pivoting or rotating about an axis that ishorizontal so that, during operations, the cameral tilt input isvertical and thus intuitive for the pilot.

In one embodiment the input device comprising paddle 134 is, aspreviously mentioned, biased to the null position using a centeringspring or other mechanism to generate a force. The magnitude of theforce may vary with the magnitude of the deflection, thus providing aforce-feedback mechanism. Passive mechanical, vibration haptic or activemechanical feedback to the user may also be incorporated. For example,haptic or tactile feedback may, optionally, be generated for the userwhen the input device leaves and/or returns to the null position.

Although the input device for commanding camera tilt need only bedisplaceable in one degree of freedom, it may also be implemented usinga multi-axis mounting that supports pivoting of a control member in onedegree of freedom that lies with a vertical plane relative to the user'sframe of reference when the base is in its normal operating positionwith respect to the user, but with its other degrees of freedom lockedor inactive.

If the target includes a camera mount with both tilt and pan capability,the input interface for cameral control input for the controller may bemodified for displacement in two degrees of freedom. The input devicewould be capable of detecting the degree of angular displacement in eachof two degrees of freedom independently, and the controller wouldrespond by generating a control input for commanding tilt and a separatecontrol input for commanding pan. In this embodiment (not shown), theinput device is mounted on the base so that one degree of freedom lieswithin a plane that is oriented vertically with respect to the user'sframe of reference when operating the controller, and a second degree offreedom is within a plane that is oriented horizontally with respect tothe user's frame of reference. Deflection up and down of a controlmember, such as joystick, causes the input to generate a rate adjustmentcommand for the camera mount to tilt the camera with respect to thetarget. Deflection from side-to-side causes the controller to generaterate adjustment command for the camera mount to pan the camera withrespect to the target. Examples of control members for a two degree offreedom input device include not only a joystick, but also a paddle,button or other structure that is constrained to move two degrees offreedom. The two degree of freedom input device would, in oneembodiment, be biased to return to a null or zero input position forboth degrees of freedom using springs or similar mechanisms. In anotherembodiment, where absolute command of the camera pan and/or tilt isdesired, the input device can be moved to the desired camera positionand released without it returning to the null position tilt and/or pan.

In still other embodiments, a camera pan control may be added to thebase. Except for its orientation, different embodiments of the pancontrol operate in the same way as the different embodiments of the tiltcontrol described above operate. However, it could, optionally, lookand/or feel differently so that it can be easily distinguished by touchand/or sight. The pan control is oriented so that deflection of the pancontrol is side-to-side or horizontal, and thus is natural, intuitive,and comfortable correspond to the pan of the camera. Thus, for example,the pan tilt control may be a paddle-type of input device like inputdevice 134 for camera tilt, except instead of being configured so thatcontrol member is deflected up and down, it is oriented to be deflectedfrom side to side to correspond with the side-to-side panning movementof the camera. However, it may be modified to generate absolute commandsfor positioning the pan of the sensor or POV, without the biasing forceto recentering it to the null position upon release.

When a pilot establishes an intent to tilt the camera to track an objectof interest, the pilot inputs a tilt command by deflecting the paddle134 and then releasing it. The controller, in response, generatescontrol input that is transmitted to the remote vehicle's onboardcontrol system over a wireless communications channel. The control inputcomprises a rate command sent to the remote vehicle or target. Theremote vehicle's onboard control system then causes the camera mount totilt the camera at a rate and for duration that is proportionate to themagnitude and duration of the deflection of the paddle. If a controlinput is available for panning, the same process can be used with aninput device that deflects from side-to-side, with the control inputbeing issued to command the remote vehicle's onboard system to pan thecamera.

The examples described above for controlling camera tilt and pan mayalso be used to steer other types of sensors—a still or video camerabeing one type of image sensor that can be attached to amotor-controlled mount on the target and rotated in one or two degreesof freedom to tilt or pan the sensor relative to the frame of referenceof the target.

Input devices for controllers of the type described above for generatingcontrol inputs to steer cameras or sensors can be characterized as beingcomprised of a control member that is supported by a mount that allowsfor the control member to be manually deflectable in one or two degreesof freedom, with the degree of deflection preferably measurable by oneor more sensors in the mounting. Examples of mounts that can be used tosupport the control members include a single or multi-axis gimbal and aball and socket joints. Examples of sensors that can be used to detectand, optionally, measure displacement of manually displaceable interfaceelements and control members include switches, inertial measurementunits, potentiometers, optical encoders, Hall effect sensors, and thelike. A processor within the controller responds to signals generated bythe sensors and generates control inputs that are transmitted by radiofrequency, optical or wired (electrical or optical) signals or acombination to the target.

Mechanisms that support measuring of angular displacement of controlmembers to indicate displacement, such as gimbals, may optionallyinclude springs for centering the control member and generating forcefeedback. Couplings or linkages that connect the joystick to a gimbalsupport, for example, could, in some embodiments, be made adjustable oradaptable to accommodate joysticks of different sizes for differentsized users.

FIGS. 2A, 2B illustrate schematically a representative example of animplementation of a two-axis gimbal mount 200 that can be used as partof an input device for generating control inputs to command a camera orsensor steering system. The two-axis gimbal mount can be used to supportsimultaneous angular displacement and measurement of the angulardisplacement in two degrees of freedom but may be adapted by locking onedegree of freedom to be used to support a control member fordisplacement in a single degree of freedom. The gimbal can be mounted ina base, such as base 114 (see FIGS. 1A-E). Its post 202 couples thegimbal mount to a control member. The control member pivots the postabout two orthogonal axes that intersect at the center of the gimbal.One axis remains fixed relative to the base and the other rotates aboutthe fixed axis. Two-axis gimbal mount 200 is a representative example ofa two-axis gimbal that has been adapted to generate to haptic feedbackupon the control member leaving and reentering a predefined nullposition for each of these two axes of rotation.

Furthermore, in an alternate embodiment in which the gimbal can belocked or blocked from rotation about one axis to allow only forrotation about one axis, the detents for generating force feedback forrotation about the locked or blocked axis could be omitted.

The gimbal is comprised of two members: one that remains fixed withrespect to base or enclosure and second member that is constrained bythe first member to rotate about a single axis or to rotate about eachof two orthogonal axes, and to otherwise restrict relative rotation ofthe first and second members around any other axis. A post is coupled tothe second member to pivot about each of the two orthogonal axes. If thesecond member is restricted to rotate only about one of the twoorthogonal axes, the post is coupled with the second member so that itis can pivot about the second axis without rotating the second member.In this particular implementation, which is intended to berepresentative, a ball 206 is mounted within a socket 208. An extension212 of the post fits within a complementary opening formed in the ball206 so that angular displacement or pivoting of the post 202 alsorotates the ball. In this example, the ball is retained within thesocket so that it can freely rotate within the socket in two degrees offreedom, about each of two axes that are mutually orthogonal to eachother, with one of the two axes remaining fixed relative to a base 209of the gimbal mount. It may, optionally, be permitted to rotate about athird mutually orthogonal axis extending through the post. The base 209is representative of a structure for mounting the gimbal, against whicha control member may react.

A cap 210 that is connected with the post extends over aspherically-shaped outer surface of the socket 208 and has acomplementary, spherical inner surface. Pivoting of the post moves thecap relative to the socket.

Although an inner surface of socket 208 can complement and supportrotation of the ball 206, the ball 206 can, in alternative embodiments,be supported for rotation about one or both mutually orthogonal axes ofrotation in other ways and by other means, including by one or moreshafts or axles that support rotation of the ball relative to thesocket. In such an alternative embodiment, the ball 206 and insidesurfaces of the socket 208 need not be spherical or complementary.

This structure could be modified, or other structures could besubstituted for the ball and socket, for a two-axis gimbal forsupporting pivoting of the post 202 in two degrees of freedom. Forexample, the complementary opening in the ball in which extension 212fits could be formed as a slot that allows the post to pivot about afulcrum located within the slot that is aligned with a first one of thetwo orthogonal axes of rotation. Thus, it could pivot without rotatingthe ball. Rotation of the ball is constrained to allow for rotation onlyabout a second one of the two orthogonal axis of rotation, which wouldthen have a fixed position relative to base 209. Deflecting the postabout the second axis of rotation would case the ball to rotate aboutthe second axis.

To detect angular deflection of the post about each of the two axes ofrotation, one or more magnets 216 are placed at the bottom of ball 206(when in the null position.) This allows a printed circuit board (PCB)218 with at least one Hall effect sensor 220 to be positioned closely todetect and measure angular displacement of the ball in up to tworotational degrees of freedom and thereby generate signalsrepresentative of the displacement. The Hall effect sensor is preferablya three-dimensional Hall effect sensor, in which case one is sufficient.One advantage to this arrangement is that the springs and the joystickare positioned higher up, keeping the bottom of the gimbal available forplacement of a Hall effect sensor. Other types of sensors could be, inother embodiments, substituted for the Hall effect sensor and magnet,including optical encoders, potentiometers, and other types of sensorsor detectors for detecting rotation of the gimbal about each of theaxes.

The same gimbal support could be adapted to be used as part of an inputdevice that moves only in one degree of freedom by locking one of thetwo axes temporarily, dynamically (based on the capabilities of theparticular target being controlled or a mode of operation of the targetor the camera), or permanently. Locking can be done physically byincorporating a structural feature that interferes with pivoting aboutone axis of rotation. For example, the socket 208 and ball 206 could beconfigured to lock the ball and socket in one axis of rotation, thusallowing relative rotation about only one of the two axes of rotation.Examples of such a lock include a pin that can be placed or selectivelyslid into and out of an interfering position or a latch that can bepivoted into and out of an interfering position, either selectively orpermanently. Alternatively, at the time of making the gimbal, acomponent that allows for movement in one degree of freedom can besubstituted with one that does not. In other embodiments the lock can beimplemented with a magnet or electromagnet that provides sufficientresistance.

The gimbal mount 200 comprises a means for generating haptic feedbackthat is mechanical and comprises at least one detent for each degree offreedom. In this example, gimbal 200 comprises at least two detents 204that cooperate with surface features on ball 206 located at the nullposition for one of the two degrees of freedom. The pair of detents andsurface features interact to generate a mechanical force that can befelt when the post enters or leaves a null position for one axis ofrotation. The other pair of detents are positioned orthogonally to thepair of detents 204 that can be seen. The detents generate a mechanicalforce feedback when entering and leaving a null or zero position whenrotating about the other axis of rotation. A single detent could be usedfor each direction of rotation, but a pair provides balance.

The detents have a rounded or spherical engaging surface that is biasedoutwardly but displaceable inwardly once the biasing force is overcome.In this example, the detents are in the form of balls that are biased bysprings 205, but other types of detents could be substituted. Only apair of detents can be seen in the figures. The detents have one or morebiasing springs mounted in a sleeve with a lip that retains the ball butallows it to extend.

A cap 210 is connected with the post 202 and comprises a cup-shapedmember with a spherically shaped inner surface that complements aspherically shaped outer surface of socket 208. It is capable ofpivoting in two degrees of freedom that correspond to the degrees offreedom in which the post 202 may pivot, the spherical inner surfacerotating about the point where the two mutually orthogonal axesintersect when the post is pivoted.

In this example, all detents 204 engage the groove 214 when the ball isrotated to the null position in both directions of rotation. Instead ofcontinuous groove 214, a small depression or dimple for each detentcould be placed at the null position for each detent. The detents areall located in the same plane, which is normal to a central axis of thepost 202. They are equally spaced at 90-degree intervals around theintersection of the central axis and the plane. When the post 202 is ina null position, one opposing pairs of detents is positioned so that thedetents in that pair are colinear along a line that is parallel to oneof the two orthogonal axes of rotation, and the other opposing pair ofdetents 204 are colinear along a line that is parallel to the other ofthe two orthogonal axes of rotation.

Instead of having the interfering surface features—the dimples orgroove—on ball 206, an alternative embodiment may use the inner surfaceof the cap 210 and the outside of the socket 208 to support the detentsand interfering surface features that interact to generate a hapticfeedback when the post is moved from a predefined null position in eachof the degrees of freedom.

A detent engaging and disengaging from the groove or other depressionprovides mechanical tactile feedback to a user at null positions in twoaxes of rotation. While the two-axis gimble mount is in the nullposition, each detent is aligned with a corresponding dimple or othertype of recess, indentation, depression, groove, or surface feature. Thesurface feature is shaped to allow the detent to extend under itsbiasing force and thus interfere with the relative movement of the capand socket. When a sufficient torque is applied by the post 202 toovercome the force created by the interference of the detent and thesurface feature, the biasing force is overcome, and the detent is pushedinward to allow the relative movement. It remains pushed inward until italigns again with a depression in the surface that allows it to extend.A deflection of the post 202 around each one of the axes of rotationwill thus be met with at least some resistance, and the resistance willbe felt as a haptic feedback to a user moving the post by moving acontrol member. Similarly, when the post 202 pivots back to a nullposition in one of the degrees of freedom the detents for that degree offreedom will extend into the recess. A user will feel the actuationforce to relax subtly as the detent passes by one side of the wall ofthe recess that forms the dimple and extends into the dimple. The usermay also feel the detent hitting the wall of the dimple on the otherside of the dimple, reinforcing the user's sense that they're back atzero. The drop off in resistance is followed by a ramp up of resistanceis the haptic cue that communicates to the user that the null positionfor that degree of freedom has been reached without having to look or tofind the null position, such as by releasing the control member andallowing it to return under a spring force to the null position. Thiscan be of advantage in many applications, particularly those in whichthe user is mobile.

Referring back to FIGS. 1A-E, controller 110 will generate controlinputs to the target in four degrees of freedom but can be adapted orconfigured to generate control in up to any number of degrees of freedomup to six degrees of freedom. Controller 110 has a first control member112 with three independent degrees of movement that is intended to begripped by a user's hand. A proximate end of control member 112 ispivotally connected to base 114 so that control member 112 can beindependently pivoted along an x-axis and independently pivoted along ay-axis. Pivoting control member 112 along the x-axis tilts the moves thedistal end of control member 112 side to side, relative to base 114.Pivoting control member 112 along the y-axis pivots the distal end ofcontrol member 112 forward and backward relative to base 114. Controlmember 112 could be made to pivot along only the x-axis or y-axis or maysimultaneously pivot around both the x and y axes. The pivotalconnection between control member 112 and base 114 may be made by anumber of pivoting type connections, such as a gimble or a ball andsocket. The controller is also movable in a third degree of freedom bytwisting it around a z-axis that is orthogonal to the x and y axes. Thefirst control member thus displaceable in three degrees of freedom, eachof which can be used to generate three, independent control inputs forcommanding three degrees of freedom of movement of the target.

A fourth degree of movement and of freedom is provided by control member116 located on a distal end of control member 112. Control member 116 inthis example is a thumb displaceable along the z-axis by operator'sthumb is placed onto the base of control member 116 and retained bythumb catch 122. Thumb catch 122 is adjustable to accommodate the sizeof a particular user's thumb so that movement of the user's thumb willbe translated into movement of control member 116. Control member 118 isa trigger style control member that may be operated by a user's indexfinger on the same hand that is gripping the first control member. Thecontrol member 116 and control member 118 are linked to move inopposition to one another generate a fourth control input. An operatoruses the operator's thumb and control member 116 to move control member116 downward by pressing down with the user's thumb. A user may movecontrol member upward by raising the user's thumb which is retained bythumb catch 122, which is connected to control member 116. An operatormay also move control member 116 upward by squeezing control member 118.The cooperation between control member 116 and control member 118 may beaccomplished in any number of ways, including using a direct, mechanicallinkage or by actuators. In alternative embodiments, control member 118can be omitted or not linked to control member 118, in which case it canbe used for other purposes.

This controller could be adapted in alternative embodiments to allow fordifferent degrees of freedom of displacement for each of its first andsecond control members. For example, the second control member can beallowed to be displaced by a user's thumb or index finger in twoadditional degrees of freedom so that the controller can be displaced insix degrees of freedom with single hand and thus generate sixindependent control inputs. In alternate embodiments, displacement ofthe third control member could be used to generate another control inputand not be linked to the second control member. Many control scenariosmay benefit from being able to provide rotational and translationalmovement using a single hand, even if fewer than all control outputs forall six degrees of freedom are required. The unused degrees of freedomcould be locked out temporarily or permanently.

Hand controller 110 also comprises display 124 on an extension 120 atthe top of the controller. The extension allows the display to be betteroriented toward the operator, so the operator may operate the handcontroller and observe display 124 at the same time. It also allows foraccommodation of second control member 116. Display 124 is used toprovide situational awareness information to the operator regarding thedevice being controlled by the hand controller. In this example, thedisplay includes a direction-to-target indicator 126 is comprised of aseries of display elements arranged in a half circular pattern thatcorresponds to the 180-degree arc directly in front of the handcontroller. Direction-to-target indicator 126 indicates the direction ofthe drone from the hand controller by illuminating the light thatcorresponds to the relative direction of the drone from the handcontroller. The display also comprised of target orientation indicator128. Target orientation indicator 128 indicates the orientation orheading of the target relative to the hand controller. A mobile,two-handed controller system like the one described above is capable ofproviding a consistent, known reference frame (stabilized by thenon-dominant hand) even while moving, e.g., walking, skiing, running,driving. For certain types of applications, for example inspection,security and cinematographic drone missions, a hand controller may bemounted on a platform that can be held or otherwise stabilized by theuser's other hand. The platform may include secondary controls and, ifdesired, a display unit. In one example, all 6-DoF inputs of acontroller, having first control member with 3-DOF of movement and asecond control member mounted to it with an additional 3-DOF ofmovement, can be reacted through the platform. With such an arrangement,this example of a control system facilitates movement through the airlike a fighter pilot with intuitive (non-deliberate cognitive) inputs.

U.S. patent application Ser. Nos. 13/797,184, 15/071,624, 15/964,064 and16/114,190, which are incorporated herein by reference for all purposes,disclose several examples of controllers that allow a single hand of anoperator to generate control inputs in more than three, and up to six,degrees of freedom (6-DoF), simultaneously and independently. Variousaspects of the single-handed controllers described in these applicationsbetter enable users, whether they are in motion or at rest (such ascomputer augmented or virtual reality gamers, pilots, hikers, skiers,security/SAR personnel, war-fighters, and others, for example) tocontrol an asset or target in physical and/or virtual three-dimensionalspace, by generating control inputs while also limiting cross-coupling(unintended motions). A controller with these features can be used toallow the controller to decouple translation (movement in physicalspace, or X, Y and Z directions) from attitude adjustments(reorientation in pitch, yaw and roll) in the control requirements ofcomputer aided design, drone flight, various types of computer games,virtual and augmented reality and other virtual and physical tasks whereprecise movement through space is required such as fixed wing and rotarywing flight, aerial refueling, surgical robotics, terrestrial and marinerobotic control, and many others. The aspects described above can beadapted for use with the embodiment described in the referencedapplications.

The base may, optionally, incorporate additional manually displaceableuser interface elements such as keys, buttons, dials, touchpads,trackpads, track balls. By locating the first control member midline tothe controller off-axis moments are reduced and making it symmetricallows for use with either hand. However, the base could be madeasymmetric. It could also be modified to allow reconfiguration foreither hand with a quick disconnect for the joystick and two mountingpoints. In other embodiments, rather than having a user hold the base,the base could be configured to be stabilized by mounting the base tothe user's body. Example of mounting points for a base on a user's bodyinclude a chest mount, a belt, and an article of clothing.

Variations may be made in the above without departing from the scope ofthe invention. While specific embodiments have been shown and described,modifications can be made by one skilled in the art without departingfrom the spirit or teaching of this invention. The embodiments asdescribed are exemplary only and are not limiting. Many variations andmodifications are possible and are within the scope of the invention.Furthermore, one or more elements of the exemplary embodiments may beomitted, combined with, or substituted for, in whole or in part, withone or more elements of one or more of the other exemplary embodiments.Accordingly, the scope of protection is not limited to the embodimentsdescribed, but is only limited by the claims that follow, the scope ofwhich shall include all equivalents of the subject matter of the claims.

What is claimed is:
 1. A controller for remotely controlling movement ofa target having a point of view (POV), the controller comprising: a baseshaped to be held by one hand of a user when moving the target, the basehaving an intended orientation with respect to the user when being heldduring operation of the remote object; a manually-displaceable controlmember mounted to the base for generating control inputs to commandmovement of the remote object; and an input device mounted on the baseand deflectable up and down on the base with respect to the intendedorientation, the deflection of the input device causing the controllerto generate a tilt command to tilt the POV.
 2. The controller of claim1, wherein input device is biased to a center position from which it canbe deflected up and down and, when released, return to the centeredposition, the center position corresponding to a null input.
 3. Thecontroller of claim 2, wherein tilt command is a rate commandproportionate to the amount of deflection.
 4. The controller of claim 2,wherein input device further comprises a paddle that extends outwardlyfrom the base.
 5. The controller of claim 1, wherein the base isconfigured to be held by one hand of a user while the second hand of theuser is gripping the control member, the base having an intendedorientation with respect to the user when being held during operation ofthe remote vehicle, wherein the input device is located on a front endof the base.
 6. The controller of claim 5, wherein the input device isbiased to a center position from which it can be deflected up and downand, when released, return to the centered position; the center positioncorresponding to a null input.
 7. The controller of claim 1, wherein theat least one control member comprises a first control member and asecond control member mounted to the first control member, the first andsecond control members having at least four degrees of freedom forgenerating control inputs for commanding movement of the remote vehiclein at least four corresponding degrees of freedom.
 8. The controller ofclaim 1, wherein the input device is further deflectable side-to-side tocause the controller to generate a camera pan command in response to aside-to-side deflection.
 9. The controller of claim 1, wherein thecontroller further comprising a rotary wheel mounted on the base, therotary wheel having a plurality of positions, each position associatedwith one or more predefined parameters for setting one or more ofparameters associated with a sensor on the target.
 10. The controller ofclaim 1, wherein the target comprises one of: a remote vehicle, on whichis mounted steerable platform for a sensor, the POV comprising theorientation of the sensor; or a virtual object, the movement of which ina virtual environment is being controlled by the control inputsgenerated by the at least one manually-displaceable control member. 11.A controller for remotely controlling a remote vehicle, on which ismounted a sensor, the controller comprising: a base shaped to be held byone hand of a user, the base having an intended orientation with respectto the user when being held during operation of the remote vehicle; atleast one manually-displaceable control member mounted on top of thebase for generating control inputs to command movement of the remotevehicle; and an input device mounted on the base and causing generationof a tilt command when deflected up and down with respect to the basewhen the base is held in its intended orientation.
 12. The controller ofclaim 11, wherein the at least one manually-displaceable control membercomprises a first control member and a second control member that ismounted to the first control member, the first and second controlmembers having at least four degrees of freedom for generating controlinputs for commanding movement of the remote vehicle in at least fourcorresponding degrees of freedom.
 13. The controller of claim 11,wherein the input device is comprised of a two-axis gimbal, the two-axisgimbal comprising, for each of two axes of rotation a detent that isaligned with a surface feature when the gimbal is in a predefined nullposition, the detent and surface feature causing haptic feedback to auser when the input device is deflected from and returned to thepredefined null position for the axis.
 14. The controller of claim 11,wherein the input device is comprised of a two-axis gimbal, and whereinthe two-axis gimbal is comprised of a base for mounting the gimbal; apost to which a control member of a controller may be coupled; a firstmember connected with the base in a fixed relationship; a second memberrotatable with respect to the first member around at least one of thetwo, intersecting axes of rotation and being constrained by the firstmember from rotating around other axes, the post being coupled with thesecond member and constrained by the second member to pivot about eachof the two, intersecting axes of rotation, the post having a nullposition at a predetermined angular displacement about each axis ofrotation.
 15. The controller of claim 14, further comprising a detentaligned with a surface feature when the angular position of the postwith respect to the first member about a first one of the axes ofrotation is in a predetermined null position, the detent and surfacefeature cooperating to cause generation of haptic feedback when the postleaves and returns to the null position, one of the detents and thesurface features coupled with the post and the other of the detents andthe surface features coupled with an outer surface of the first member.16. A controller for remotely controlling movement of a target having apoint of view (POV), the controller comprising: a base shaped to be heldby one hand of a user when moving the target, the base having anintended orientation with respect to the user when being held duringoperation of the remote object; a first, manually-displaceable controlmember mounted to a base for generating control inputs to commandmovement of the remote object, and a second control member mounted tothe first control member, the first and second control members having atleast four degrees of freedom for generating control inputs forcommanding movement of the remote vehicle in four corresponding degreesof freedom; and an input device mounted on the base and deflectable upand down on the base with respect to the intended orientation togenerate a tilt command to tilt the POV, the input device located on afront of the base in a position that enables a finger on the handholding the base to deflect the input device, the deflection of theinput device generating of the tilt command.
 17. The controller of claim16, wherein the input device is biased to a center position and returns,when released after deflection, to the centered position.
 18. Thecontroller of claim 16, wherein input device is further deflectable fromside to side to generate a pan command.
 19. The controller of claim 16,wherein the tilt command is a rate command proportionate to the amountof deflection.
 20. The controller of claim 16, wherein the input devicefurther comprises a paddle or a post that extends outwardly from thebase.
 21. The controller of claim 18, wherein the pan command is a ratecommand proportionate to the amount of deflection.
 21. A controller forremotely controlling movement of a target with a sensor mounted on asteerable platform to adjust its orientation, the controller comprising:a base; at least one manually-displaceable control member mounted to abase for generating control inputs to command movement of the remoteobject; an input device mounted on the base, the input device comprisinga rotary wheel having a plurality of positions, each of the plurality ofpositions associated with one or more predefined parameters, at leastone of the predefined parameters for each of the two or more of theplurality of positions being selected from the group consistingparameters relating to the sensor and to a responsiveness of thesteerable platform to commands for adjustment of the orientation of thecamera.
 22. The controller of claim 21, wherein the group consists ofparameters relating to the sensor.
 23. The controller of claim 22,wherein the sensor is an image sensor in a camera, and the parametersrelating to the sensor consist of camera a focal length, shutter speed,aperture, video frame rate, white balance, and exposure compensation.24. The controller of claim 21, wherein the group consists of parametersrelating to the responsiveness of the steerable platform to commands foradjustment of the orientation of the camera.
 25. The controller of claim21, further comprising: at least one manually-displaceable controlmember mounted to the base for generating control inputs to commandmovement of the remote object; and an input device mounted on the baseand deflectable up and down on the base with respect to the intendedorientation, the deflection of the input device causing the controllerto generate a tilt command for the steerable platform to tilt thesensor.